Exponential bounds on the estimation error are given for the plug-in estimator of weighted areas under the ROC curve. The bounds hold for single score functions and uniformly over classes of functions, whose complexity can be controlled by Gaussian or Rademacher averages. The results justify learning algorithms which select score functions to maximize the empirical partial area under the curve (pAUC). They also illustrate the use of some recent advances in the theory of nonlinear empirical processes.

If X is unambiguous we write ˆ µ = ˆµ (X ). A summary of notation in tabular form is given in the supplement.

One contribution seems to have been in defining a surrogate functional g (line 166) that replaces the \mu(0) term in a denominator term with an arbitrary parameter c and then using a uniform convergence bound over values of c to ensure that estimation does take place even if c is replaced with its actual value of \mu(0). Another contribution seems to be in fine tuning the proof technique used to prove Proposition 5. The main contribution is a proof for obtaining generalization bound for weighted areas under the ROC curve for Lipschitz weight functions.

This is a theoretical paper that has received relatively good reviews. However, two of the reviewers only increased their scores from 5 to 6 in order to reduce the divergence and help form a consensus (in the discussions), but neither was really convinced about the quality of the paper. Unfortunately, the highest scoring reviewer was also the least confident. I read the paper myself and I find that it has some merits --- it seems theoretically solid, but I have a slight tendency towards saying that it may be a better fit at ALT/AISTATS/COLT, and it is unclear if the NeurIPS community will benefit from knowing these results. Nevertheless, regardless of the final outcome, the authors are encouraged to improve the readability of their paper through (it is currently somewhat dense for the average reader).

Exponential bounds on the estimation error are given for the plug-in estimator of weighted areas under the ROC curve. The bounds hold for single score functions and uniformly over classes of functions, whose complexity can be controlled by Gaussian or Rademacher averages. The results justify learning algorithms which select score functions to maximize the empirical partial area under the curve (pAUC). They also illustrate the use of some recent advances in the theory of nonlinear empirical processes.